OOCOBBHT IBSOMB 



BO 095 872 



IB 001 053 



&0TRO9 
TITIB 

INSTITOTION 

PUB OATB 
BOTE 



BDKS PBICB 
0ESCBIPT06S 



IDBBTIPIERS 



ABST9ACT 



Teikin, Sanford 

Raking Sen8« of Co8t-E££ectl¥«nt88 and Benefit-Cost 
Analyses. 

Besearch for Better Schools, Inc., Philadelphia, 
Pa. 

19 Apr 7i| 

15p.; Paper presented at the Annual Meeting of the 
Rational Society for Performance and Instruction 
(Riasi Beach, Florida, April 1974) 

RF-10.75 HC-11.50 PLDS POSTAGE 

*Cost Effectiveness; Costs; ^Decision Raking; 

♦Educational Accountability; Educational 

Adsinistration; Educational Assesssent; Estisated 

Costs; *Ranagesent by Objectives; Perforsance 

Specifications; *Progras Effectiveness; Resource 

Allocations 

Benefit Cost Analysis 



Although tvo econosic sethods, cost effectiveness and 
benefit-cost analysis, are frequently sentioned as useful tools for 
educational decision saking, only one, cost effectiveness, has 
potential for saking a contribution to this field. A benefit-cost 
analysis tries for each alternative to seasure benefits and costs, 
which are then discounted to yield a present value. Onofrtunately, in 
educational systess benefits cannot be seasured, and the idea of a 
discount has no seaninq. Rowever, a cost effectiveness analysis say 
have application to education since it cospares the different costs 
of alternatives against the different perforsance levels on 
objectives. A decision criterion sust be able to relate and seasure 
all the perforsance of difference alternatives. Rost educational 
organi2ation8 do not have the data for a cost effectiveness analysis, 
yet the thinking and judgsents on relative value and perforsance are 
Bost desirable. (WR) 



MAKING SENSE OF COST -EFFECTIVENESS AND BENEFIT-COST ANALYSES 

Sanford Temkin 
Research for Better Schools, Inc. 



My foiml academic education has been primarily in economics and 
statistics. In 1965, I first began to puzzle over ways to apply economic 
methods to the problems of education. It was then that the concepts and 
methods of benefit-cost and cost -effectiveness analyses became familiar 
to me. 

I was asked to speak to you today about making sense out of cost- 
effectiveness and benefit-cost analyses. While that is a foiroidable ob- ' 
jective, nonetheless, I believe that we can do just that. 

Before we examine each of the methods it will be useful to draw a 
distinction bet^veen two kinds of analytic remedies. 

Conceptual An alytic Models and Practical Analytic Tools 
I would like now to draw a distinction between conceptual analytic 
models and practical analytic tools. A conceptual analytic model describes 
a set of information requirements and a method to transfoim these infonna- 
tion requirements into a useful management decision tool. Conceptual anal- 
ytic models often exist several years before practical applications are made. 
These models are inportant for they provide a general framework to guide those 
who will ultimately perform the first practical applications. 
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A practical analytic tool describes a set of infoimation requirements 
which are feasible - that is, information requirements that are obtainable 
or could easily be obtainable and are within the existing state of the art 
of measurement technology -- and a method to transform these information 
requirements into a useful management decision tool. 

The essential distinction, then, is that for an analytic tool to be* 
considered as practical it must be technically possible to obtain the in- 
formation required. 

Now we shall consider the conceptual analytic models of benefit-cost 
analysis and cost-effectiveness analysis. After that we shall examine 
these models by asking ourselves if they are more appropriately classified 
as conceptual or practical. 

As we shall see both approaches have roots deriving from methods de- 
veloped by economists. 

BENEFIT-COST ANALYSIS 

Benefit -cost analysis derives from work, primarily in the field of 
water resource development, some of which was done as early as the 1920's. 
Methodologically it aims at the selection of one or more projects from a 
pool of alternatives. Each alternative is viewed as a capital investment 
project with the analysis focusing upon benefits and costs. 

IVhen a banker evaluates a potential investment, he considers the stream 
of incomes and costs over the lifetime of the venture. He knows that receiv- 
ing $1000 ten years from now is worth less than receiving $1000 today because 
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he can invest the $1000 he receives today and have close to $2000 in 10 
years. Benefit -cost analysis evaluates a public-sectot^^investment just 
like the banker evaluates a private -sector capital investment. 

Let us now consider the infonnation requirements and how this in- 
formation is assembled into a decision criterion. 
Information Requirements 

1. For each alternative, we need an estimate of benefits to be 
derived for each year. Ihe symbol B^^ refers to the benefits 
for alternative 1 in time period t. 

2. For each alternative, we need an estimate of costs to be incur- 
red for each year. The symbol C^^ refers to the costs for al- 
ternative 1 in time period t. 

3. For each alternative, we need a definite number of years for 
which the project alternative will be considered. The symbol 
N refers to the number of years benefits and costs are to be 
explicitly included in the analysis for a given alternative. 

4. For all alternatives a discount rate must be selected to use 
as a way of translating future benefits and costs into present 
day tenns. The discount rate is a "reverse" interest rate. 
The symbol ^ refers to the discount rate. 

The Decision-Criterion 

The tern decision-criterion refers to how the decision -maker sorts 
alternatives into those which are acceptable and those which are not ac- 
ceptable. 
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Nov. we a5seml)lc the information so as to be able to calculate the 



present value fjr each alternative. Tlie symbol V^^^ is the present value 
of alternative 1. That is, 

N 



This calculation is performed for each alternative and the V's are com- 
pared. Higher \^'s are preferred. 
The .Acid Test: Feasibility 

Let us consider the application of this approach to education 
more specifically the public schools. I4any controversies and arguinents 
arise out of attempts to operationally define the information require- 
ments under any set of realistic conditions. Essentially, we are con- 
fronted by three main problems in using bene "^it-cost analysis as a basis 
for school district decisions. 

I . The superintendent of schools is continually bombarded by im- 
mediate pressures . He is forced to show results now. In 
a well-balanced conmunity, the superintendent may be able to 
trade-off some of the pressure for present-period results for 
the promise of expected improvements in the future. But only 
the nost farseeing of conmunities will allow costly future-or- 
iented programs to be initiated totally at the expense of the 



]<hile I have selected the school district as a focal point, much 
of the argument extends to such diverse consideration as govemment invest- 
ment in educational R § D, university investment in new curricula, state 
level investment in new initiatives suggested through legislation and the 
like. 




piesci t. In pl'innin". at the national level, some fom of ben- 
efit-cost analysis may be desirable, but the superintendent, 
as a result of many years of "non -planning" ..t the local level, 
is not today afforded this basic view of planning for his schools. 

2. Beaefit-cost analysis requires that both benefits and costs be 
translated into coninon units (typically dollars) . This is usually 
an unmanageable problem, since the benefits of education are, in 
general , both intangible and incommensurable . IVhile it may be 
I'ossible to reduce benefits to a common unit;* it is conceptually 
iiuprobable that this unit would be dollars. Ratio of benefits to 
costs could be utilized but this change introduces the methodolog- 
ical idic '?>'ncracies of ratios into the already murkey waters. 

3. A discount rate must, according to tiie requirements of the method, 
be chosen to adjust future benefits and costs to present-value 
terms . How does one select a discount rate for the intangibles 
of education? This is a problem of considerable consequence to 
the analysis because studies in the water resource domain and other 



Ullis B. Page has been working on this problem. See for instance: 

Ellis B. Page and Thomas F. Breen III. "Educational Values for 
^feasurement Technology: Some Theory and Data" in Frontiers of Educational 
Management and Information Systems - 1973. Proceedings of an invitational 
Conference on the Occasion of the Dedication of the Lindquist Center for 
Measurement, loh'a City, f^ril 6-7, 1973. Boston: Houghton-Mifflin, 1973. 

Also see, Ellis B. Page. "Seeking a Measure of General Educational 
Advancement: The Bentee," Journal of Educational Measurement, Spring, 1972. 
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domains have consistently demonstrated that choice of a discoiait 
rate parameter (e.g., 41 versus 5t) is sufficient to alter the 
rankings of V's for the alternatives being considered. Add to 
this the 1974 inflation- interest rate fluctuation realities and 
what do you have? 
Sunmary for Benefit-Cost Analysis 

While benefit-cost analysis exhibits potential as a way of concep- 
tual thinking and as a way of viewing problems, it offers little of prac- 
tical significance, in ray judgment, to those of us who want to make better, 
more informed educational decisions. 

As a vague, vogue, somewhat general notion, it has been bantered about, 
especially by funding agencies when they are in a cutting posture -- ••The 
benefit -cost picture here does not look very good.^^ This, of course, means 
that the fundee must, in order to contradict the conclusion, find some data- 
based way of demonstrating that the benefit-cost picture is , indeed, good. 
Struggle as you may, there is no way to provide comter evidence. > 

COST-EFFECTIVENESS ANALYSIS " 
Cost-effectiveness analysis derives, principally, from work done in 
the Department of Defense under Secretary McNamara in the early 1960 's. Many 
of the technical problems of developing military systems were analyzed within 
some fom of effectiveness/cost framework. 

Methodologically, it aims at the selection of one or more alternatives 
each of v/hich has been designed to meet one or more objectives. It is a 
natural substitute for benefit-cost analysis for situations in which benefits 
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are iaconimcniurable aiid inappropriate for ddllar valuation. 

The method is usually applied with one of two perspectives: 

1. For a given level of effectiveness, find the alternative (s) 
that minimizes cost outlays. ^ 
or 2. For a given level of cost outlay, find the altemative(s) that 

maxLuizes effectiveness. 
Information Requirements 

In 1970, Research for Better Schools published a monograph that I 
wrote on this subject. At the time my interests were to identify the 
kinds of information that a school district would need to (1) assess where 
it was and (2) know how to proceed from there. 

I developed a framework -.liich went from the over-simplified to the 
comi)Icxities of schools. Tlie analyses in cu^'i of the cases developed 
within the framework focused upon infoimation requirements. Let us look 
at the essentials of what I found. (See Figure 1 on the following page). 

Perhaps we can slowly walk through the framework. It begins with 
what is perhaps the simplest decision case of all a decision-maker 
wants to select one alternative from a set of alternatives; he is un- 
constrained by dollars and the frame of reference if j-priori (before 
the fact) -- that is, he is also unconstrained by any ongoing activities. 
This case is what I referred to as "a Utopian R § D situation." 

The analysis continues and I will now highlight some of what we 

find. 



Sec reference 3 under Cost -effectiveness. 



FIGURE 1 

A C\SE CLASSIFICATION OF LOGICALLY RELATED DECISION PRQBLEKB 
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STRUaURH 



CASE 



1. Single objective with 
set of proposed plans 
(acti vi ty -des igns) 
for achieving the ob- 
jective; one plan is 
to be selected. 

2. Single objsctive with 
set of activity-de- 
signs; one to be 
selected. 

3. Evaluation of Cases 
1 and 2. 



4. Single objective with 
set of tasks; several 
to be ! ;lected as a 
program package. 

5. Single objective with 
set of tasks; s'^veral 
to be selected as a 
program package. 

6. Evaluation of Cases 
4 and 5. 

7. Multiple objectives 
with multiple activ- 
ities; several to be 
selected. 

8. Multiple objectives 
with multiple activ- 
ities; several to be 
selected. 



DECISION 
FRA\eiDRK 



A priori 



RESOURCE 
LEVEL 



Unlimited 



CCKflENT 



A Utopian research and 
development problem. 



A priori 



Limited 



A posteriori Known 



A priori Uhlimited 



A priori 



Limited 



A post *riori Known 



The constraint limits ad- 
missible altematives; but 
still no incentive to econ- 
omize. 

A perfoimance evaluation 
involving a partition of 
outcome space. 

A more conqslex version of 
Case 1; still Utopian. 



Ihe constraint limits ad- 
missible alternatives as 
in Case 2; still no in- 
centive to economize. 

A more conplex version of 
Case 3. 



A priori Unlimited A much more conplex version 

of Cases 1 M; the relative 
weight of objectives beoomes 
in|)ortant. 



A priori Limited 



9. Evaluation of Case 8. A posteriori Known 



The general cost -effective- 
ness case; the only case 
meeting the necessary and 
sufficient conditions for 
cost -effectiveness analysis, 

Ihe general program evalua- 
tion case; provides cost- 
effectiveness evaluations 
for present year and inputs 
for next year's budget. 



Sooner or later we end up with Case 9 which is the evaluation of 
an ongoing system with multiple objectives and multiple activities de- 
signed to meet these objectives. Case 8 is the situation in which the 
findings of Case 9 are used to allocate resources for the future. 

Some other highlights of possible interest are: 

1. The first genuine incentive to economize (by economize I mean 
to get more for the dollar -- not to reduce costs in the strict 
sense of the word) is Case 8 in which we have multiple obj^- 
tives (and thus multiple outcomes that are inportant) and a 
constrained budget.. Budgetary constraints by themselves do 
not necessarily result in more economical operations nor 

do multiple objectives necessarily result in better allocations 
of resources. 

2. Public sector organizations are usually not rewarded for find- 
ing a cheaper way to do something. Sometimes in fact there 

may be disreward because there £re indications that 'you did not 
need all that money." This is sad. 

3. Benefit -cost analysis comes in at Case 8 if i ..s in anywhere 
(this, again, is because of the feasibility impediments). 

The Decision Criterion 

It is somewhat difficult to specify a precise decision criterion ^ 
here but I will try. In doing this Tmust simplify some rather conplex 
relationships. 

The overall thrust of the approach is to help an ongoing organiza- 
tion achieve more of its objectives given the dollars it has to work with. 
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It is i:iade performance scores from con|)onent activities and value 

assignments made to these activities. Thus we have 



^1 * effectiveness for activity 1 



N 



and 



2 P|V. « sum of the effectiveness scores 
i-1 for all N activities. 



What we try to do is rearrange activities in such a way that we can pro- 
duce the largest EPv for the amouit of money available. 
Let us examine two sketches to help our thinking 



Overall 
Objective 



Sketch #1 
A View of the Organization 

Programs and Projects 



*A1 



*A2 



• • • 



a 



61 



X2 



a, 



'C3 



Hie symbol 0^ refers to overall objective A. The nunbers refer to 
programs and projects. An activity, e.g., a^^ refers to program I's con- 
tribuf^a to overall objective B. 

To conduct the appropriate analyses it is necessary that each over- 
all objective be weighted to reflect -.iiportance (this of course brings ,. 
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us to Ai. totl s 'whose values"). Further it is necessary that the 
activities cont iijuting to a particular overall objective be assigned 
values. And lastly, it s necessary that a performance measure (some- 
times more than one performance measure) be assigned to each activity, 
ideally on a scale from 0 to 100. With these pieces of data we can ar- 
rive at a measure of effectiveness. 

This part of the analysis will tell us v^ere we have been -- that 
is, how well the organization is now performing with respect to its pres 
ent overall objectives. 

To see the effects of altemative changes it is necessary, however, 
to introduce costs. 

Jketch #2a 

Ihe Effectiveness -Cost Point for a Single Activity 



Effectiveness 



El 



^ Point 1 




Point 1 tells us that if is spent it will produce E^ results. 
'Jhfortunately this does not tell us what to do -- even if and E^ were 
actual figures we still would not know vfhat to do. TTius, the infoima- 
tion we ha\^ about effectiveness and cost as a result of our present year's 
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operations is not sufficient to tell us anything about what to do next. 

IVhat do we need to know lii order to make decisions? First we need 
a curve for each act'^'ity similar to that presented in Sketch #2b. 

Sketch #2b 

Point 1 




Cost ^1 



With this information we can now see that point 1 is a very good 
level of funding for the activity represented in the sketch. The effec- 
tiveness axis, remember, reflects both value assignments and perfoimance. 
Now if we have curves such as this for each activity (standardization 
, being achieved through value assigkiments and costs) , we can allocate a 
budget and consider the potential effects of that allocation on organ- 
izational objectives. 

Where is the rub? First, it is highly unlikely that any organiza- 
tion has the full range of data to allow someone to construct the sketch 
in 2b. Secondly, then, selected points on the curve must be "estimated" 
using expert judgment. We must delicately balance objective and sub- 
jective data. All of this opens many new questions -- yet when we do 
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not explicitly u; j tlicse data our decisions, nevertheless, implicitly 
presuppose the data. Even if the data are very soft, it is reasonable, 
in my opinion, to exi-uct tha' lecis ion -makers provide explicit judgments 
in an open maniier. 
Ihc Acid Test: Feasibilit y 

This approach is, ver>' generally, implementable but with difficulty 
because, after all, this is a new way of thinking. For school districts 
to implement such an approach there is need for extensive training and 
given the tenor of the times, a bottom-up implementation strategy may 
be required. By "bottom-up" I mean that teachers and principals should 
work with overall objectives, perfoimance measures and figure out how 
well they are doing. Then, as they get feeling for the data that de- 
scribe present performance they can push for changes needed to stpport 
the classroom. This push will create the need for cost infoimation and 
tlic^.cost and effectiveness can get together. 
Summary 

I have tried to draw for you some frameworks within the area of 
applying economic -based methods to education. There are other methods 
that I have not considered because the agenda was already packed vlth 
material. The very nature of tlic material is highly technical and as 
such we may have spent too much time on technicalities at the expense 
of overall understanding. There is, however, a certain amount of tech- 
nical appreciation that is prerequisite to overall understanding and I 
hope that I have struck an appropriate balance. 
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biases are strong as to the relative applicability of the two 
methods to education, benefit -cost analysis just is not feasible while 
cost -effectiveness can, I believe, help us sharpen our judgments and, 
perhaps, enable us to be more effective in the decisions we make. 

How cllVci .vely have we our objective of making sense out of 
the methods? vs we recall, effectiveness as a measure has two contrib- 
uting elenionts -- relative value and performance. For those of you who 
attended this session, relative value must have been reasonably high. 
Hie final as.^essments of performance I mast leave to you. 
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